Zero-shot event detection using semantic embedding

ABSTRACT

Zero-shot content detection includes building/training a semantic space by embedding word-based document descriptions of a plurality of documents into a multi-dimensional space using a semantic embedding technique; detecting a plurality of features in the multimodal content by applying feature detection algorithms to the multimodal content; determining respective word-based concept descriptions for concepts identified in the multimodal content using the detected features; embedding the respective word-based concept descriptions into the semantic space; and in response to a content detection action, (i) embedding/mapping words representative of the content detection action into the semantic space, (ii) automatically determining, without the use of training examples, concepts in the semantic space relevant to the content detection action based on the embedded words, and (iii) identifying portions of the multimodal content responsive to the content detection action based on the concepts in the semantic space determined to be relevant to the content detection action.

GOVERNMENT RIGHTS

This invention was made with government support under contract no.D11PC20066 awarded by the Intelligence Advanced Research ProjectsActivity by way of the Department of the Interior National BusinessCenter. The Government has certain rights in this invention.

BACKGROUND

Every minute, hundreds of hours of video are uploaded to video archivalsites such as YouTube. Developing methods to automatically understandthe events captured in such large volume of videos is necessary andmeanwhile challenging. One of the important tasks in this direction isevent detection in videos. A main objective of this task is to determinethe relevance of a video to an event based on the video content (e.g.,feeding an animal, birthday party).

Typically, the classification of complex events captured in video byautomated computer analysis currently relies on machine-learning-trainedclassifiers that attempt to recognize such events based on low-levelvisual features extracted from the video. These classifiers typicallymust be trained with many labelled video examples for each one of thecomplex events that is to be detected.

SUMMARY

Embodiments of the present principles generally relate to zero-shotcontent detection.

In accordance with one embodiment an automated content detection systemfor semantic retrieval of multimodal content includes one or morenon-transitory machine accessible storage media and instructionsexecutable by one or more processors to cause a computing system tobuild a semantic space by embedding word-based document descriptions ofa plurality of documents into a multi-dimensional space using a semanticembedding technique; detect a plurality of features in the multimodalcontent by applying feature detection algorithms to the multimodalcontent; determine respective word-based concept descriptions forconcepts identified in the multimodal content using the detectedfeatures; embed the respective word-based concept descriptions into thesemantic space; and in response to a content detection action, such as aconcept query or an event query, (i) embed words representative of thecontent detection action into the semantic space, (ii) automaticallydetermine, without the use of training examples, concepts in thesemantic space relevant to the content detection action based on theembedded words, and (iii) identify portions of the multimodal contentresponsive to the content detection action based on the concepts in thesemantic space determined to be relevant to the content detectionaction.

In some embodiments, the content detection system implementsdistributional semantic embedding and the plurality of documents includea plurality of text-based documents. In some embodiments the pluralityof text-based documents include at least one of online news sources,online dictionaries or online encyclopedic material and the word-baseddocument descriptions of the plurality of documents describe concepts inthe plurality of documents.

In some embodiments, the respective word-based concept descriptions areembedded into the semantic space based on a location of the word-baseddocument descriptions of the plurality of documents in the semanticspace. In some embodiments, the concepts identified in the multimodalcontent include at least one of audio content, video content and textcontent. In some embodiments, the words representative of the contentdetection action are embedded into the semantic space based on alocation of the word-based document descriptions of the plurality ofdocuments in the semantic space and corresponding word vectors for thewords representative of the content detection action.

In some embodiments, concepts in the semantic space relevant to thecontent detection action are determined using a relative location of theword-based concept descriptions and the words representative of thecontent detection action in the semantic space and a similarityfunction. In various embodiments, a number of results to be retrieved asa result of the content detection action can be limited by a distancemeasure threshold or can be limited to a number of results, for example,by a user input.

In some embodiments, the computing system is further configured toautomatically deliver the identified portions of the multimodal contentto a user for display.

In some embodiments, a content detection system for a zero-shot eventquery of multimedia content includes one or more non-transitory machineaccessible storage media and instructions executable by one or moreprocessors to cause a computing system to determine a semantic space byembedding a first set of words retrieved from a plurality of trainingdocuments into a multi-dimensional space using a distributional semanticembedding technique; detect a plurality of semantic features inmultimedia content by applying semantic feature detection algorithms tothe multimedia content; identify, as a second set of respective words,concepts in the multimedia content using the detected features; embedthe second set of respective words representative of the identifiedconcepts into the semantic space based on an embedding location of thefirst set of words; and in response to an event query, (i) embed a thirdset of words representative of the event query into the semantic spaceand (ii) automatically determine at least one concept of the multimediacontent relevant to the event query based on the embedded second set andthird set of words.

In some embodiments, the computing system is further configured toidentify portions of the multimedia content responsive to the eventquery based on the concepts in the semantic space determined to berelevant to the event query and deliver the identified portions of themultimedia content to a user for display. In some embodiments, theconcepts identified in the multimedia content include at least one ofaudio content, video content and text content.

In some embodiment, a computing device for zero-shot content detectionof multimodal content, includes a processor; and a memory coupled to theprocessor, the memory having stored therein a multimedia contentunderstanding module, a natural language generator module and a semanticspace generator module. The computing device includes instructionsexecutable by the processor to configure the computing device to build asemantic space by embedding word-based document descriptions of aplurality of documents into a multi-dimensional space using a semanticembedding technique; detect a plurality of features in the multimodalcontent by applying feature detection algorithms to the multimodalcontent; determine respective word-based concept descriptions forconcepts identified in the multimodal content using the detectedfeatures; embed the respective word-based concept descriptions into thesemantic space; and in response to a content detection action, (i) embedwords representative of the content detection action into the semanticspace, (ii) automatically determine concepts in the semantic spacerelevant to the content detection action based on the embedded words,and (iii) identify portions of the multimodal content responsive to thecontent detection action based on the concepts in the semantic spacedetermined to be relevant to the content detection action, without theuse of training examples.

In some embodiments the computing device includes a search requestmodule and a user interface device for enabling an initiation of acontent detection action. In some embodiments the computing deviceincludes at least one of an optical character recognition system togenerate text transcripts of text in the multimodal content and anautomatic speech recognition system to generate text transcripts ofspeech in the multimodal content. In some embodiments the computingdevice is further configured to deliver the identified portions of themultimodal content to a user for display. In some embodiments, theconcepts identified in the multimodal content include at least one ofaudio content, video content and text content.

In some embodiments a method for zero-shot content detection includesbuilding a semantic space by embedding word-based document descriptionsof a plurality of documents into a multi-dimensional space using asemantic embedding technique; detecting a plurality of features in themultimodal content by applying feature detection algorithms to themultimodal content; determining respective word-based conceptdescriptions for concepts identified in the multimodal content using thedetected features; embedding the respective word-based conceptdescriptions into the semantic space; and in response to a contentdetection action, (i) embedding words representative of the contentdetection action into the semantic space, (ii) automatically determiningconcepts in the semantic space relevant to the content detection actionbased on the embedded words, and (iii) identifying portions of themultimodal content responsive to the content detection action based onthe concepts in the semantic space determined to be relevant to thecontent detection action, without the use of training examples.

In some embodiments, the concepts identified in the multimodal contentinclude at least one of audio content, video content and text content.In some embodiments the method further includes delivering theidentified portions of the multimedia content to a user for display. Insome embodiments, the content detection action comprises at least one ofa concept query or an event query.

Other and further embodiments in accordance with the present principlesare described below.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentprinciples can be understood in detail, a more particular description ofthe principles, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments in accordance with the present principles and aretherefore not to be considered limiting of its scope, for the principlesmay admit to other equally effective embodiments.

FIG. 1 depicts a high level block diagram of a content detection systemfor zero-shot content detection in accordance with an embodiment of thepresent principles;

FIG. 2 depicts a graphical representation of a trained semantic space inaccordance with an embodiment of the present principles;

FIG. 3 depicts a 3D PCA visualization of an event query and thecorresponding 20 most relevant concepts in a trained space as determinedusing a similarity function in accordance with an embodiment of thepresent principles;

FIG. 4A depicts a flow diagram of a method for zero-shot contentdetection in accordance with a general embodiment of the presentprinciples;

FIG. 4B depicts a flow diagram of a method for zero-shot event query inaccordance with a more specific embodiment of the present principles;

FIG. 5 depicts a high level block diagram of a computing device in whichthe content detection system of FIG. 1 can be implemented in accordancewith an embodiment of the present principles.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. The figures are not drawn to scale and may be simplifiedfor clarity. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

Embodiments of the present principles generally relate to methods,apparatuses and systems for zero-shot content detection using semanticembedding. While the concepts of the present principles are susceptibleto various modifications and alternative forms, specific embodimentsthereof are shown by way of example in the drawings and are described indetail below. It should be understood that there is no intent to limitthe concepts of the present principles to the particular formsdisclosed. On the contrary, the intent is to cover all modifications,equivalents, and alternatives consistent with the present principles andthe appended claims. For example, although embodiments of the presentprinciples will be described primarily with respect to visual concepts,such teachings should not be considered limiting. Embodiments inaccordance with the present principles can be applied to audio conceptsand other such concepts as described herein.

Reference will be made to multimedia and multimodal content throughoutthis disclosure. It should be noted that the terms can be usedinterchangeably herein to refer to content that has multimediacomponents, such as text audio and video, and/or multimodal components,such as speech, graphics, and gestures.

Search and retrieval of videos for arbitrary events using onlyfree-style text and unseen text, in particular, has been an unrealizedgoal in computational video and multi-media understanding. This isreferred to as “zero-shot event detection” mainly because there is nopositive exemplar videos to train a detector. Due to the proliferationof videos, especially consumer-generated videos (e.g., YouTube),zero-shot search and retrieval of videos has become an increasinglyimportant issue.

Embodiments in accordance with the present principles are directed atautomatically retrieving relevant concepts and events inmultimedia/multimodal content by leveraging information from semanticembedding, such as distributional semantic embedding. An advantageousresult is the ability to use a detection event, such as an event textquery (i.e. just the event title like “birthday party” or “feeding ananimal”), to retrieve a list of video(s) and or video clips (i.e., aranked list of video clips), frames, segments, and the like based ontheir content. For example, in some embodiments in accordance with thepresent principles, information from a semantic space, such as adistributional semantic space, trained using a plurality of content,such as a large text corpus, is leveraged to embed/map concepts, contentdetection queries and multimedia/multimodal content to the same space,where similarities can be directly estimated. In some embodiments inaccordance with the present principles, various similarity metrics areimplemented in a distributional semantic space to enable contentdetection (e.g., event retrieval) based on, for example, (a) concepts,(b) ASR, and (c) OCR in videos. The unified framework is capable ofembedding (a), (b), and (c), above, into a same semantic space enablinghigh speed content detection/retrieval results, which makes itapplicable to both a large number of multimedia/multimodal content andconcepts. In some embodiments, concepts, video information and eventqueries are embedded into a distributional semantic space whosedimensionality is independent of the number of concepts.

As described above, in some embodiments in accordance with the presentprinciples, a semantic space is trained on a large text corpus. Invarious embodiments the semantic space is a statistical distribution andis itself computer-generated through statistical analysis of largecorpuses of documents, such as online collections of news stories,and/or online encyclopedic material like Wikipedia.

For example, in some embodiments, a plurality of terms can be receivedfrom textual documents such as Wikipedia and Google news. The terms caninclude phrases (i.e., groups of words) and independent words, and caneven be a single word. The terms received can be projected into at leastone ontology source, to determine a normalized set of conceptsrepresented by the received terms. An ontology source in this contextrepresents a data source that describes the relationships of particularterms to concepts (e.g., the words used to describe a particular conceptin an encyclopedia), and can further relate the described concepts toone another. Exemplary ontology sources that can be used for thispurpose include community-generated content such as generalencyclopedias (e.g., Wikipedia®), directories (e.g., the Open DirectoryProject), and topic-specific encyclopedias (e.g., the Internet MovieDatabase). Domain-specific ontologies and/or dictionaries can also beused as ontology sources, whether general or topic-specific (e.g.,medical dictionaries and legal dictionaries). Community-generatedontology sources are typically the result of iteration, modification,and refinement by a group of community members, such that the resultingdata represents a consensus of the community on the meanings andrelationships of the particular represented terms, topics, and concepts.

The word(s) and/or phrases (i.e., first set of word(s) and/or phrases)representative of the concepts/topics can be thereby embedded into acommon semantic space in terms of textual language/words. Thus, invarious embodiments, through a combined use of, for example, (a) videoclassifiers for familiar, lower-level concepts in multimodal conceptstrained in terms of low-level visual features, and (b) the embeddedsemantic space representing relationships between textual words anddescriptors of concepts and complex events, an innovative, powerful,multimodal classification capability is provided to facilitate usersperforming free-text searching for video clips depicting complex eventsof interest.

In accordance with various embodiments of the present principles, asemantic space is built/trained using a distributional semantic model.An example of a distributional semantic model that can be used fortraining a semantic space, M_(s), in accordance with the presentprinciples can be found in Mikolov, T.; Chen, K.; Corrado, G.; and Dean,J. 2013a. “Efficient estimation of word representations in vectorspace”; in ICLR and Mikolov, T.; Sutskever, I.; Chen, K.; Corrado; G.S.; and Dean, J. 2013b. “Distributed representations of words andphrases and their compositionality”; in NIPS, the teachings of which areboth incorporated herein by this reference in their entireties. For suchspaces, words and their context in billions of documents along withnegative examples of words out of context are used to train an embeddingspace in which mapped vectors in context are neighbors while mappedvectors out of context are not. This method produces a high-dimensionalspace, typically 300-D, in which related words are closer than unrelatedwords.

The trained semantic space can be denoted by M_(s) and the vectorizationfunction that maps a word to M_(s) space can be denoted as vec(•). Thedimensionality of the real vector returned from vec(•) is denoted by M.The models learn a vector for each word, w_(n), such thatp(w_(n)|(w_(i−L), w_(i−L+1), . . . , w_(i+L−1), w_(i+L))) is maximizedover the training corpus of documents, in which, for the example above,2×L is the context window size. The term above captures the context ofw_(n) within the context of words that occur before w_(n) and those thatoccur after w_(n) as a probability distribution. As such, a similaritybetween vec(w_(i)) and vec_((i)) is high if the words co-occurred manytimes in, for example, a context of the size 2×L in the trainingtext-corpus (i.e., semantically similar words share similar context).Based on the trained M_(s) space, it is defined how to embed a contentdetection action, such as an event query e, and concepts, c. That is, anevent query for concept based retrieval can be denoted by e_(c), whilequery keywords for OCR and ASR can be denoted by e₀ and e_(a),respectively. As such, a total event query can be represented ase={e_(c), e₀, e_(a)}. Each of e_(c), e_(a), and e₀ is a set of one ormore words. Each of these words can be directly embedded into the M_(s)manifold by a vec(•) function. Accordingly, the sets of word vectors foreach of e_(c), e_(a), and e₀ can be represented as θ(e_(c)), θ(e_(a)),and θ(e₀), collectively denoted as θ(e). Regarding the embedding ofconcepts, c, each concept c*ϵc is defined by its name and optionallysome related keywords. As such, the corresponding word vectors are thenused to define an embedding location, θ(c*), for a respective concept inthe M_(s) space (described in further detail below).

In accordance with embodiments of the present principles, a plurality ofsemantic features are detected in multimedia content by applyingsemantic feature detection algorithms to the multimedia content.Concepts in the multimedia content are then identified as naturallanguage word(s) and/or phrases (i.e., a second set of words) using thedetected features in the multimedia content. The term “concepts” can beused herein to describe what is traditionally referred to as a conceptin multimedia content, a speech transcript, where the speech transcriptcan be generated by, for example, an automated speech recognition (ASR)module of the content detection system 110 and/or the computing device100, and a text transcript, where the text transcript can be generatedby, for example, an optical character recognition (OCR) module of thecontent detection system.

For example, referring to FIG. 1, as illustrated, a content detectionsystem 110 in accordance with some embodiments of the present principlescan be embodied in a computing device 100, an example of which is shownin FIG. 5, described below. In FIG. 1, an embodiment of the contentdetection system 110 is shown in the context of an environment that canbe created during the operation of the content detection system 110(e.g., a physical and/or virtual execution or “runtime” environment).The illustrative content detection system 110 is embodied as a number ofmachine-readable instructions, modules, data structures and/or othercomponents, which can be implemented as computer hardware, firmware,software, or a combination thereof.

The content detection system 110 of FIG. 1 executes computer visionalgorithms, including machine learning algorithms, semantic reasoningtechniques, and/or other technologies to, among other things, in anautomated fashion, identify, understand, and describe concepts that aredepicted in a multimedia content input 102 that can, in one embodiment,comprise a collection of user content. As described in more detailbelow, the illustrative content detection system 110 can, among otherthings, identify features, concepts and events in multimedia content102, such as user stored multimedia content, and enables users toquickly and easily locate particular events and concepts in lengthyand/or large volumes of multimedia content, such as still pictures andvideo footage, without the typical need for extensively trained machinelearning of the complex events. The content detection system 110 cangenerate a natural language (NL) description (e.g., a second set ofword(s) and/or phrases) of the multimedia content input 102, whichdescribes the multimedia content of the input 102 in a manner that canbe used for embedding, at least, low-level concepts and, in someembodiments, higher-level events detected in the multimedia content intoa semantic space, such as a distributional semantic space (described inmore detail below).

More specifically, the content detection system 110 can be used toautomatically detect and describe concepts and events in the multimediacontent (also referred to as multimodal content, video and/or videocontent herein). As used herein, “concepts” can also refer to, amongother things, lower-level events that are made up of multiple“constituent” people, objects, scenes, audio, text and/or activitiesthat make up more complex events. For example, some concepts related toa more complex event, such as a birthday party, can include theactivities of singing, blowing out candles, opening presents, and eatingcake. Similarly, some related concepts of a more complex event such as achild acting out an improvisation can include people smiling, laughing,dancing, drawing a picture, and applause and other concepts related tothe complex event of a group activity relating to a political issue,sports event, or music performance can include a group of people walkingor standing together, a person holding a sign, written words on thesign, a person wearing a t-shirt with a slogan printed on the shirt, andhuman voices shouting. The activities that make up a concept are notlimited to visual features. Rather, “activities” of a concept as usedherein may refer to, among other things, visual, audio, and/or textfeatures, which may be detected by the content detection system 110 inan automated fashion using a number of different algorithms and featuredetection techniques, as described in more detail below. Stated anotherway, an activity as used herein may refer to any semantic element of themultimedia input 102 that, as determined by the content detection system110, evidences a concept.

As used herein, “multimedia content/input” can refer to, among otherthings, a collection of digital images, a video, a collection of videos,or a collection of images and videos (where a “collection” includes twoor more images and/or videos) some or all of which can include an audiocomponent and text component. References herein to a “video” can referto, among other things, a relatively short video clip, an entirefull-length video production, or different segments within a video orvideo clip (where a segment includes a sequence of two or more frames ofthe video). Any video of the multimedia input 102 can include or haveassociated therewith an audio soundtrack and/or a speech transcript,where the speech transcript may be generated by, for example, anautomated speech recognition (ASR) module of the content detectionsystem 110 and/or the computing device 100. Any video or image of theinput 102 can include or have associated therewith a text transcript,where the text transcript can be generated by, for example, an opticalcharacter recognition (OCR) module of the content detection system 110and/or the computing device 100. References herein to an “image” mayrefer to, among other things, a still image (e.g., a digital photograph)or a frame of a video (e.g., a “key frame”).

A multimedia content understanding module 104 of the content detectionsystem 110 of FIG. 1 is embodied as software, firmware, hardware, or acombination thereof. The multimedia content understanding module 104applies a number of different feature detection algorithms 130 to themultimedia input 102, and in some embodiments, using a multimediacontent knowledge base 132, and generates one or more of a speechtranscript 107, a text transcript 108 and natural language conceptdescriptions (e.g., a set of respective words identifying the concepts)106 based on the output of the algorithms 130. The multimedia knowledgebase 132 is embodied as software, firmware, hardware, or a combinationthereof (e.g., as a database, table, or other suitable data structure orcomputer programming construct). The illustrative multimedia contentunderstanding module 104 executes different feature detection algorithms130 on different parts or segments of the multimedia input 102 to detectdifferent features, or the multimedia content understanding module 104executes all or a subset of the feature detection algorithms 130 on allportions of the multimedia input 102.

Some examples of feature detection algorithms and techniques, includinglow-level, mid-level, and complex event detection and recognitiontechniques, are described in a commonly owned patent application, Chenget al., U.S. Utility patent application Ser. No. 13/737,607(“Classification, Search, and Retrieval of Complex Video Events”); andalso in Chakraborty et al., U.S. Utility patent application Ser. No.14/021,696, filed Sep. 9, 2013 (“Recognizing Entity Interactions inVisual Media”), Chakraborty et al., U.S. Utility patent application Ser.No. 13/967,521, filed Aug. 15, 2013 (“3D Visual Proxemics: RecognizingHuman Interactions in 3D from a Single Image”), Han et al., U.S. Pat.No. 8,634,638 (“Real-Time Action Detection and Classification”), Eledathet al., U.S. Pat. No. 8,339,456 (“Apparatus for Intelligent andAutonomous Video Content and Streaming”), all of SRI International andeach of which is incorporated herein by this reference. Additionally,technologies for visual feature detection and indexing are disclosed inSawhney, Harpreet S. et al., U.S. Utility patent application Ser. No.14/452,237 (“Multi-Dimensional Realization of Visual Content of an ImageCollection”), which is incorporated herein by this reference.

For example, in at least one embodiment, both static and dynamiclow-level visual features are detected. Static visual features includefeatures that are extracted from individual key-frames of a video at adefined extraction rate (e.g., 1 frame/second). Some examples of staticvisual feature detectors include Gist, SIPT (Scale-Invariant PeatureTransform), and colorSIFT. The Gist feature detector can be used todetect abstract scene and layout information, including perceptualdimensions such as naturalness, openness, roughness, etc. The SIFTfeature detector can be used to detect the appearance of an image atparticular interest points without regard to image scale, rotation,level of illumination, noise, and minor changes in viewpoint. ThecolorSIFT feature detector extends the SIFT feature detector to includecolor keypoints and color descriptors, such as intensity, shadow, andshading effects.

Dynamic visual features include features that are computed over x-y-tsegments or windows of the video. Dynamic feature detectors can detectthe appearance of actors, objects and scenes as well as their motioninformation. Some examples of dynamic feature detectors include MoSIFT,STIP (Spatio-Temporal Interest Point), DTFHOG (Dense Trajectory basedHistograms of Oriented Gradients), and DTF-MBH (Dense-Trajectory basedMotion Boundary Histogram). The MoSIFT feature detector extends the SIFTfeature detector to the time dimension and can collect both localappearance and local motion information, and identify interest points inthe video that contain at least a minimal amount of movement. The STIPfeature detector computes a spatial-temporal second-moment matrix ateach video point using independent spatial and temporal scale values, aseparable Gaussian smoothing function, and space-time gradients. TheDTF-HoG feature detector tracks two-dimensional interest points overtime rather than three-dimensional interest points in the x-y-t domain,by sampling and tracking feature points on a dense grid and extractingthe dense trajectories. The HoGs are computed along the densetrajectories to eliminate the effects of camera motion). The DTF-MBHfeature detector applies the MBH descriptors to the dense trajectoriesto capture object motion information. The MBH descriptors represent thegradient of optical flow rather than the optical flow itself. Thus, theMBH descriptors can suppress the effects of camera motion, as well.However, HoF (histograms of optical flow) may be used, alternatively orin addition, in some embodiments.

Concept detection is then performed by, for example, the multimediacontent understanding module 104 based on the low-level features. In oneembodiment, the multimedia content understanding module 104 uses one ormore concept classifiers to analyze the low-level features and derivetherefrom semantic elements, which represent the higher-level concepts.

For example, in some embodiments, concept detectors are applied to thefeatures to identify the atomic, concept-level semantic elements (e.g.,actions, scenes, objects, actors, audio, text) associated with thelow-level features. In some embodiments, static features, dynamicfeatures, and BoW representations defined over codebooks of the featuresare used as inputs to the concept detectors. Concept classifiers (e.g.,binary SVM classifiers) can be used to classify the features accordingto their concept type. A concept detector is defined or selected foreach concept type (suitable concept detectors are publicly available)and applied to the features associated with the concept type. Forexample, an action concept detector might determine whether a videofeature likely depicts an “animal eating” or a “person dancing,” while ascene concept detector can determine whether the video feature likelydepicts a “church” or a “kitchen” and an object concept detector maydetermine whether the video feature likely depicts a “wheel” or “food.”

Generally speaking, a concept detector takes an input x, applies aconcept detection algorithm to x, and outputs a detection confidenceindicating the likelihood that x depicts the concept that the conceptdetector is designed to detect. In the case of concept detectors thatdetect actions, the input x is a short segment of the video. For conceptdetectors that detect scenes, objects, or actors, the input x is akey-frame or a series of key-frames sampled from the video. Once aconcept is identified, a natural language (word) description can bedetermined for the concepts, using, for example, a natural languagegenerator. That is, in various embodiments in accordance with thepresent principles, concepts in the multimedia content are identifiedusing the detected features. The concepts can be identified as a secondset of respective word(s) and/or phrases representative of theidentified concepts.

With reference to FIG. 1, in some embodiments, to identify concepts inthe multimedia content 102, the illustrative multimedia contentunderstanding module 104 accesses one or more feature models 134 and/orconcept models 136 and optionally event models 138. The feature models134, the concept models 136 and optional event models 138 are embodiedas software, firmware, hardware, or a combination thereof, e.g., aknowledge base, database, table, or other suitable data structure orcomputer programming construct. The models 134, 136, 138 correlatesemantic descriptions of features, concepts and, optionally, events withinstances or combinations of output of the algorithms 130 that evidencethose features and concepts and events. For example, the feature models134 may define relationships between sets of low level features detectedby the algorithms 130 with semantic descriptions of those sets offeatures (e.g., “object,” “person,” “face,” “ball,” “vehicle,” etc.).Similarly, the concept model 136 can define relationships between setsof features detected by the algorithms 130 and higher-level “concepts,”such as people, objects, actions and poses (e.g., “sitting,” “running,”“throwing,” etc.). Similarly, the event models 138 can definerelationships between sets of concepts and events. The semanticdescriptions of features and concepts and events that are maintained bythe models 134, 136, and optionally 138 can be embodied as naturallanguage descriptions and/or structured data. In some embodiments inaccordance with the present principles, a mapping 140 of the knowledgebase 132 defines relationships between various combinations of features,concepts, events, and activities. The concept can be determined usingsemantic reasoning in connection with the knowledge base 132 and/or themapping 140. To establish relationships and associations, the contentdetection system 110 can utilize, for example, a knowledgerepresentation language or ontology.

The mapping 140 of the knowledge base 132 links activities withconcepts, so that, once a concept is determined, the understandingmodule 104 can determine activities that are associated with the conceptand look for those activities in the input 102. A natural languageconcept description (e.g., set of word(s) and or phrases) 106 for atleast the identified concepts can then be generated using, for example,a natural language generator module 119 of the content detection system110. The mapping 140 can establish one-to-one, one-to-many, ormany-to-many logical relationships between the various concepts andactivities in the knowledge base 132. In general, the mapping 140 andthe various other portions of the knowledge base 132 can be configuredand defined according to the requirements of a particular design of thecontent detection system 110 and/or the computing device 100 (e.g.,according to domain-specific requirements).

The illustrative content detection system 110 includes a number ofsemantic content learning modules 152, including feature learningmodules 154, a concept learning module 156 and, optionally, an eventlearning module 158. The learning modules 152 execute machine learningalgorithms on samples of multimedia content (images, video and/or audio)of an image/video collection 150 and create and/or update portions ofthe knowledge base 132. For example, the learning modules 152 can beused to initially populate and/or periodically update portions of theknowledge base 132. The feature learning modules 154 analyze sampleimages and videos from the collection 150 and populate or update thefeature models 134. For example, the feature learning modules 154 can,over time or as a result of analyzing portions of the collection 150,algorithmically learn patterns of computer vision algorithm output thatevidence a particular feature, and update the feature models 134accordingly. Similarly, the concept learning modules 156 and eventlearning modules 158 can, over time or as a result of analyzing portionsof the collection 150, algorithmically learn combinations of low levelfeatures that evidence particular concepts, and algorithmically learncombinations of concepts and higher level features that evidenceparticular events and update the concept models 136 and event models138, accordingly.

The illustrative learning modules 152 analyze portions of theimage/video collection 150 to determine criteria, to identify featuresand/or concepts and/or events for inclusion in the mapping 140, toidentify activities that are associated with the features or concepts.For example, the event learning modules 152 can identify a new feature,concept, event or activity for inclusion in the mapping 140 based on thefrequency of occurrence of certain features and/or concepts and/orevents in the collection 150. The learning modules 152 can also identifymultimodal concept markers including “non-visual” characteristics ofinput videos such as object motion, changes in motion patterns, changesin camera position or camera motion, amount or direction of cameramotion, camera angle, audio features (e.g., cheering sounds or speech).

The video collection 150 refers generally to one or more bodies ofretrievable multimedia digital content that can be stored, for example,in a memory of the computing device 100 and/or other computing systemsor devices. The video collection 150 can include images and/or videosstored remotely at Internet sites such as YOUTUBE and INSTAGRAM, and/orimages/videos that are stored in one or more local collections, such asstorage media of a personal computer or mobile device (e.g., a “cameraroll” of a mobile device camera application). In any case, images/videosin the collection 150 need not have been previously tagged withmeta-data or other identifying material in order to be useful to thecontent detection system and/or the computing device 100. The contentdetection system and/or the computing device 100 can operate onimages/videos 150 and/or multimedia input 102 whether or not it has beenpreviously tagged or annotated in any way. To the extent that any of thecontent in the collection 150 is already tagged with descriptions, anyof the learning modules 152 can learn and apply those existingdescriptions to the knowledge base 132.

The semantic space generator module 114 of FIG. 1 uses a plurality oftraining content to train a semantic space, such as a distributionalsemantic space as described above. In some embodiments in accordancewith the present principles, the training content 112 includes textbased documents, which can include online collections of news stories,and/or online encyclopedic material like Wikipedia, which can beretrieved by or communicated to the content detection system 110. Inother embodiments, such text based documents can be stored locally andcan be communicated to the semantic space generator module 114 fortraining the distributional semantic space. For example, in oneembodiment, the collection 150 includes more than just images and videosand the training documents can be retrieved from the collection 150.

In other embodiments in accordance with the present principles, thetraining content can include audio content for which audio transcriptscan be generated using, for example, automated speech recognition (ASR).Such transcripts can be communicated to the semantic space generatormodule 114 as words or phrases for training the semantic space. In otherembodiments in accordance with the present principles, the trainingcontent can include video content including text for which texttranscripts can be generated using, for example, optical characterrecognition (OCR). Such text transcripts can be communicated to thesemantic space generator module 114 as word(s) and/or phrases fortraining the distributional semantic space.

As will be described in further detail below, once the distributionalsemantic space is trained, multimedia content to be searched, such asthe multimedia content 102, can be map into the semantic space. In theembodiment of FIG. 1, the semantic space generator module 114 embeds atleast one of the natural language concept description 106, the speechtranscript 107, and the text transcript 108 into the distributionalsemantic space as word(s) and/or phrases. The respective words and/orphrases representative of the multimedia content are embedded into thedistributional semantic space by the semantic space generator module 114(see FIG. 2). That is, in various embodiments in accordance with thepresent principles, the respective words (i.e., second set of respectivewords) representative of at least the identified concepts (and in otherembodiments, the speech transcript and text transcript) are mapped intothe semantic space based on an embedding location of the word(s) and orphrases (i.e., first set of words) embedded in the semantic space as aresult of the training of the semantic space.

The search request module 118 of FIG. 1 enables a content detectionaction/search request, such as an event query or concept query, to beinitiated for retrieving and/or identifying content of the multimediacontent 102. In some embodiments, a content detection action isinitiated using a human-computer interface including one or more inputdevices (e.g., a touchscreen, keyboard, virtual keypad, microphone,etc.) (not shown). In one embodiment, the content detection action canbe configured as natural language word(s) and/or phrases (i.e., a thirdset of words). In other embodiments, the content detection action can beconfigured as speech and can be converted to word(s) and/or phrasesusing automated speech recognition (ASR).

As will be described in further detail below, the word(s) and/or phrasesrepresentative of the content detection action (i.e., a third set ofwords) are mapped into the distributional semantic space by, forexample, the semantic space generator module 114. In some embodiments inaccordance with the present principles, in response to a contentdetection action, word(s) and/or phrases representative of the contentdetection action are mapped into the distributional semantic space. Insome embodiments in accordance with the present principles, the wordsrepresentative of the content detection action are embedded into thesemantic space based on a location of word-based document descriptionsof the plurality of documents in the semantic space and correspondingword vectors for the words representative of the content detectionaction.

Content relevant to the content detection action and/or event query isdetermined based on the position/location of the word(s) and/or phrasesrepresentative of the content detection action and the word(s) and/orphrases representative of the concepts of the multimedia content in thesemantic space. That is, at least one concept of the multimedia contentrelevant to the content detection action is automatically determinedbased on the mapped word(s) and/or phrases representative of the contentdetection action (third set of words) and the word(s) and/or phrasesrepresentative of the concepts of the multimedia content (second set ofwords) in the semantic space and a similarity function (described ingreater detail below).

As described in further detail below, in some embodiments in accordancewith the present principles, word(s) and/or phrases representative ofconcepts in the multimedia content relevant to the content detectionaction are determined using, for example, a similarity function, suchas, in one embodiment, a nearest neighbor function, and a distancemeasure. Word(s) and/or phrases identified as a result of the contentdetection action can be presented as, for example, a 3D PCAvisualization of the relevant word(s) and/or phrases (see FIG. 3).

In various embodiments, video clips, frames, etc., of the multimediacontent identified by the word(s) and/or phrases in the semantic spacerelevant to the word(s) and/or phrases representative of the contentdetection action and/or event query are returned as a result of thecontent detection action and/or event query. In some embodiments, suchvideo clips, frames, etc., can be presented to an originator of theevent query, or some other user, on a display.

In some embodiments, the search request module 118 can optionallyfurther enable the defining of search parameters. For example, in oneembodiment, a user can enter a score to be used as a threshold for thereturned results.

In accordance with the present principles, given an arbitrary eventquery, e, and a video, v, in one embodiment, the objective is to modelp(e|v), which represents the probability of occurrence of an event, e,given a video, v. Initially, representations of an event query, e, aconcept set, c, and the video, v, are defined. In one embodiment, anunstructured event title is used to represent an event query for conceptbased retrieval in accordance with the present principles. In suchembodiments, additional terms specifically for ASR or OCR basedretrieval can be used. As described above, a few-keyword event query forconcept based retrieval can be denoted by e_(c), while query keywordsfor OCR and ASR can be denoted by e₀ and e_(a), respectively. As such, atotal event query can be represented as e={e_(c), e₀, e_(a)}.

A total concept set can be denoted as c, which can include visualconcepts, c_(v), and audio concepts, c_(d), with c={c_(v), c_(d)}. Invarious embodiments, the visual concepts can include object, scene andaction concepts. In addition, the audio concepts can include acousticrelated concepts like water sound. For purposes of ease ofunderstanding, the embodiments described herein will be describedprimarily with reference to visual concepts for concept-based retrieval(e.g., c_(d)=0).

Each member, c_(i) E c can be defined as the i^(th) concept in c. Invarious embodiments, c_(i) can be defined by the i^(th) concept's nameand optionally some related keywords. As such, c={c₁, . . . c_(N)} isthe set of concept definitions, where N is the number of concepts.

For zero-shot purposes in accordance with various embodiments of thepresent principles described herein, a video, v, can be defined by atleast three pieces of information, which include video OCR, denoted byv₀, video ASR, denoted by v_(a), and video concept representationdenoted by v_(c). In one embodiment, v₀ and v_(a) represent the detectedtext in OCR and speech in ASR, respectively, in the video. The videoconcept based representation, v_(c), can be defined according toequation one (1) which follows:v _(c)=[p(c _(i) |v),p(c ₂ |v), . . . ,p(c _(N) |v)]  (1)where p(c_(i)|v) represents a conditional probability of concept c_(i)given video v. Hereinafter p(c_(i)|v) is denoted as v_(c) ^(i).

In various embodiments of zero-shot event detection in accordance withthe present principles, one goal is to recognize complex events invideos without the need for training examples and based on themultimedia content of the video, including still-image concepts likeobjects and scenes, action concepts, OCR, and ASR. Although trainingexamples are not needed to recognize complex events (or optionallyconcepts embedded in the semantic space) in accordance with the presentprinciples, it should be noted that classifiers for other, lower-levelconcepts/features can be machine learned using training examples asdescribed above.

Given video components, v={v_(c), v₀, v_(a)}, one goal is to computep(e|v) by embedding both the event query, e, and information of thevideo, v, of different modalities (v_(c), v₀, v_(a)) into adistributional semantic space, where a relevance of v to e can bedirectly computed. Specifically, one approach is to model p(e|v) as afunction, F of θ(e), ψ(v_(c)), θ(v₀), and θ(v_(a)), which represents thedistributional semantic embedding of e, v_(c), v₀, and v_(a),respectively, into a semantic space (also referred to as a trainedsemantic space, distributional semantic manifold, M_(s) space, and thelike, throughout this disclosure), for which the embedding can becharacterized according to equation two (2), which follows:p(e|v)αF(θ(e),ψ(v _(c)),θ(v ₀),θ(v _(a)))  (2)

In various embodiments, the stop words, such as “the”, “an”, “and” etc.,are removed from e, v₀, and v_(a) before the embedding, θ(•).

FIG. 2 depicts a graphical representation of a semantic space, M_(s), inaccordance with an embodiment of the present principles. As depicted inFIG. 2, video components and an event query are embedded into a manifoldspace in accordance with a distributional semantic embedding model. Inthe embodiment of FIG. 2, the video components include an ASR componentof the audio track of the video, an OCR component of the visual track ofthe video, and audio and visual concepts of the video. As depicted inFIG. 2, an exemplary event query (e.g., “feeding an animal”) is alsoembedded into the manifold space.

To determine a relevance of concepts to an event query, a similarityfunction is defined between θ(c*) and θ(e_(c)) as s(θ(e_(c)), θ(c*). Ina first embodiment in accordance with the present principles, a versionof s(X, Y) is defined where the sets X and Y are initially pooled by asum operation. The sum pooling operation is denoted on a set by anoverline. For instance, X=Σ_(i)x_(i) and Y=Σ_(i) y_(j), where x_(i) andy_(j) are the word vectors of the i^(th) element in X and the j^(th)element in Y, respectively. Then, a cosine similarity between X and Y iscomputed. This similarity function can be denoted by s_(p)(⋅, ⋅) andcharacterized according to Equation three (3), which follows:

$\begin{matrix}{{s_{p}\left( {X,Y} \right)} = {\frac{\left( {\sum\limits_{i}\; x_{i}} \right)^{T}\left( {\sum\limits_{j}y_{j}} \right)}{{{\sum\limits_{i}x_{i}}}{{\sum\limits_{j}y_{j}}}} = \frac{{\overset{\_}{X}}^{T}\overset{\_}{Y}}{{\overset{\_}{X}}{\overset{\_}{Y}}}}} & (3)\end{matrix}$

FIG. 3 depicts a 3D PCA visualization of an event query for “Grooming ofan Animal” and the corresponding 20 most relevant concepts in a trainedM_(s) space as determined using the similarity function, s_(p)(⋅, ⋅).That is, FIG. 3 depicts embedding of the event query and the relevantconcept sets, θ(e_(c)=“Grooming of an Animal”) and each of θ(c_(i)) forthe most relevant 20 concepts as represented by their correspondingpooled vectors (θ(e_(c)) and θ(c_(i))∀i) normalized to unit length underL2 norm. As depicted in FIG. 3, the relevant concepts include aweighting factor (i.e., strength of association) which is representativeof the relevance of the respective concept with respect to the eventquery. The video clips, frames, etc., representative of the conceptsdepicted in FIG. 3 can be returned to and displayed for an originator ofthe event query.

In other embodiments in accordance with the present principles, s(X, Y)is defined as a similarity function between the X and Y sets. In suchembodiments, for robustness, a percentile-based Hausdorff point setmetric was used, in which a similarity between each pair of points iscomputed by the cosine similarity. Such a similarity function can bedenoted by s_(t)(X, Y) and characterized according to equation four (4),which follows:

$\begin{matrix}{{s_{t}\left( {X,Y} \right)} = {\min{\left\{ {{\overset{I\%}{\min_{j}}{\max_{i}\frac{x_{i}^{T}y_{i}}{{x_{i}}{y_{i}}}}},{\overset{I\%}{\min_{i}}{\max_{j}\frac{x_{i}^{T}y_{i}}{{x_{i}}{y_{i}}}}}} \right\}.}}} & (4)\end{matrix}$In one embodiment, I=50% (i.e., median).

In one operational embodiment, p(e|v) is decomposed into p(e_(c)|v),p(e₀|v), and p(e_(a)|v), which reduces a solution to deriving p(e_(c)|v)(concept based retrieval), p(e₀|v) (OCR based retrieval), and p(e_(a)|v)(ASR based retrieval) in the M_(s) space. With regards to derivingp(e_(c)|v) (concept based retrieval), concepts, as used herein, arerepresented in the M_(s) space by linguistic meanings that havecorresponding detection functions given the video, v. In accordance withembodiments of the present principles, M_(s) space can be represented asa space of meanings defined by a training text-corpus, where only sparsepoints in that space have corresponding visual detection functions giventhe video, v, which are the concepts, c, (e.g., “blowing a candle”). Forzero shot event detection in accordance with embodiments of the presentprinciples, these sparse points are exploited using the informationcaptured by s(θ(e_(c)), θ(c_(i)ϵc) in M_(s) space. As such, in someembodiments, p(e_(c)|v) is derived from a probabilistic perspectivestarting from marginalizing p(e_(c)|v) over the concept set c accordingto equation five (5), which follows:p(e _(c) |v)αΣ_(c) _(i) p(e _(c) |c _(i))p(c _(i) |v)αΣ_(ci) s(θ(e_(c)),θ(c _(i)))v _(c) ^(i)  (5)where p(e|c_(i))∀i are assumed to be proportional tos(θ(e_(c)),θ(c_(i))).

From a semantic embedding perspective, each video, v, is embedded intoM_(s) space by the set ψ(v_(c))={θ_(v)(c_(i))=v_(c) ^(i)θ(c_(i)), ∀c_(i)ϵc}, where v_(c) ^(i)θ(c_(i)) is a set of the same points in θ(c_(i))scaled with v_(c) ^(i). ψ(v_(c)) can then directly compared withθ(e_(c)) according to equation six (6), which follows:p(e _(c) |v)αΣ_(ci) s(θ(e _(c)),θ(c _(i)))v _(c) ^(i) αs′(θ(e _(c)),ψ(v_(c))={θ_(v)(c _(i)),∀c _(i) ϵc})   (6)where s′(θ(e_(c)), ω(P(c|v)=Σ_(i)s(θ(e_(c)), θ(e_(c)), θ_(v)(c_(i)) ands(⋅, ⋅) can be replaced by s_(p) (⋅, ⋅), s_(t) (⋅, ⋅), or othersimilarity measures in M_(s) space.

In one embodiment, only θ_(v)(c_(i)) are included in ψ(v_(c)) such thatc_(i) is among the top R concepts with highest p(e_(c)|c_(i)). This isassuming that the remaining concepts are assigned p(e_(c)|c_(i))=0 whichmakes those items vanish. In such embodiments, only a few conceptdetectors need to be computed for on v, which results in a computationaladvantage.

With respect to estimating p(e₀|v) (OCR based retrieval), and p(e_(a)|v)(ASR based retrieval) in the M_(s) space, both v₀ and v_(a) can bedirectly embedded into the M_(s) space since both are sets of words. Assuch, in one embodiment in accordance with the present principles,p(e₀|v) and p(e_(a)|v) can be modeled according to equation seven (7),which follows:p(e ₀ |v)αs _(d)(θ(e _(c)),θ(v ₀)),p(e _(a) |v)αs _(d)(θ(e _(a)),θ(v_(a)))  (7)where s_(d)(X, Y)=Σ_(ij)x_(i) ^(T) y_(j).

The similarity function described directly above can be more appropriatefor ASR/OCR text since the applications normally contain more textcompared with a concept definition. In one embodiment in accordance withthe present principles, results of e_(a) and e₀ are automaticallyaugmented with the nearest words to an event query/title in the M_(s)space using cosine similarity before retrieval. As such, relevant wordsthat might appear in v₀ and/or v_(a) are automatically retrieved.

FIG. 4A depicts a flow diagram of a method for zero-shot contentdetection in accordance with a general embodiment of the presentprinciples. The method 400 begins at step 402 during which a semanticspace is built by embedding word-based document descriptions of aplurality of documents into a multi-dimensional space using a semanticembedding technique. The method 400 can proceed to 404.

At 404, a plurality of semantic features are detected in multimodalcontent by applying semantic feature detection algorithms to themultimodal content. As described above, in various embodiments, one ormore static and dynamic feature detectors and semantic feature detectionalgorithms can be applied to the multimedia content to determine contentfeatures in the multimedia content. The method 400 can proceed to 406.

At 406, respective word-based concept descriptions are determined forconcepts identified in the multimodal content using the detectedfeatures. The method 400 can proceed to 408.

At 408, the respective word-based concept descriptions are embedded intothe semantic space. The method 400 can proceed to 410.

At 410, in response to a content detection action, (i) wordsrepresentative of the content detection action are embedded into thesemantic space, (ii) concepts in the semantic space relevant to thecontent detection action are automatically determined, without the useof training examples, based on the embedded words, and (iii) portions ofthe multimodal content responsive to the content detection action areidentified based on the concepts in the semantic space determined to berelevant to the content detection action. The method 400 can be exited.

FIG. 4B depicts a flow diagram of a method for zero-shot event query inaccordance with a more specific embodiment of the present principles.The method 450 begins at step 412 during which a semantic space isdetermined by embedding a first set of words retrieved from a pluralityof training documents into a multi-dimensional space using a semanticembedding technique. The method 450 can proceed to 414.

At 414, a plurality of semantic features are detected in multimediacontent by applying semantic feature detection algorithms to themultimedia content. The method 450 can proceed to 416.

At 416, concepts in the multimedia content are identified as a secondset of respective words using the detected features. The method 450 canproceed to 418.

At 418, the second set of respective words representative of theidentified concepts in the multimedia content are embedded/mapped intothe semantic space based on an embedding location of the first set ofwords. The method 450 can proceed to 420.

At 420, in response to an event query (i) a third set of wordsrepresentative of the event query are embedded/mapped into the semanticspace and (ii) at least one concept of the multimedia content relevantto the event query is determined based on the embedded/mapped second setand third set of words. The method 450 can be exited.

Referring now to FIG. 5, a simplified high level block diagram of anembodiment of the computing device 100 in which the content detectionsystem 110 can be implemented is shown. While the computing device 100is shown as involving multiple components and devices, it should beunderstood that in some embodiments, the computing device 100 canconstitute a single computing device (e.g., a mobile electronic device,laptop or desktop computer) alone or in combination with other devices.

The illustrative computing device 100 can be in communication with oneor more other computing systems or devices 542 via one or more networks540. In the embodiment of FIG. 5, illustratively, a portion 110A of thecontent detection system 110 can be local to the computing device 510,while another portion 1106 can be distributed across one or more othercomputing systems or devices 542 that are connected to the network(s)540. Similarly, a portion 150A of the illustrative video collection 150can be local to the computing device 100 while another portion 150B candistributed across one or more of the other computing systems or devices542. In other embodiments, however, the content detection system 110and/or the video collection 150 can be located entirely on a singlecomputing device. In some embodiments, portions of the content detectionsystem 110 can be incorporated into other systems or interactivesoftware applications. Such applications or systems can include, forexample, operating systems, middleware or framework software, and/orapplications software. For example, portions of the content detectionsystem 110 can be incorporated into or accessed by other, moregeneralized search engine(s) or intelligent assistance applications.

The illustrative computing device 100 of FIG. 5 includes at least oneprocessor 512 (e.g. a microprocessor, microcontroller, digital signalprocessor, etc.), memory 514, and an input/output (I/O) subsystem 516.The computing device 100 can be embodied as any type of computing devicesuch as a personal computer (e.g., desktop, laptop, tablet, smart phone,body-mounted device, etc.), a server, an enterprise computer system, anetwork of computers, a combination of computers and other electronicdevices, or other electronic devices.

Although not specifically shown, it should be understood that the I/Osubsystem 516 typically includes, among other things, an I/O controller,a memory controller, and one or more I/O ports. The processor 512 andthe I/O subsystem 516 are communicatively coupled to the memory 514. Thememory 514 can be embodied as any type of suitable computer memorydevice (e.g., volatile memory such as various forms of random accessmemory). In the embodiment of FIG. 5, the I/O subsystem 516 iscommunicatively coupled to a number of hardware components and/or othercomputing systems including one or more user input devices 518 (e.g., atouchscreen, keyboard, virtual keypad, microphone, etc.), and one ormore storage media 520.

The storage media 520 may include one or more hard drives or othersuitable data storage devices (e.g., flash memory, memory cards, memorysticks, and/or others). In some embodiments, portions of systemssoftware (e.g., an operating system, etc.), framework/middleware (e.g.,application-programming interfaces, object libraries, etc.), the contentdetection system 110A, and/or the video collection 150A reside at leasttemporarily in the storage media 520. Portions of systems software,framework/middleware, the content detection system 110A and/or the videocollection 150 can also exist in the memory 514 during operation of thecomputing device 100, for faster processing or other reasons.

The one or more network interfaces 532 can communicatively couple thecomputing device 100 to a local area network, wide area network, apersonal cloud, enterprise cloud, public cloud, and/or the Internet, forexample. Accordingly, the network interfaces 532 can include one or morewired or wireless network interface cards or adapters, for example, asmay be needed pursuant to the specifications and/or design of theparticular computing device 100.

The other computing device(s) 542 can be embodied as any suitable typeof computing device such as any of the aforementioned types of devicesor other electronic devices. For example, in some embodiments, the othercomputing devices 542 can include one or more server computers used tostore portions 150B of the video collection 150.

The computing device 100 can further optionally include an opticalcharacter recognition (OCR) system 528 and an automated speechrecognition (ASR) system 530. It should be understood that each of theforegoing components and/or systems can be integrated with the computingdevice 100 or can be a separate component or system that is incommunication with the I/O subsystem 516 (e.g., over a network). FIG. 5further depicts a search request module 118, which althoughillustratively depicted as a component of the computing device 100, inother embodiments in accordance with the present principles can be acomponent not part of the computing device 100.

The computing device 100 can include other components, subcomponents,and devices not illustrated in FIG. 5 for clarity of the description. Ingeneral, the components of the computing device 100 are communicativelycoupled as shown in FIG. 5 by signal paths, which may be embodied as anytype of wired or wireless signal paths capable of facilitatingcommunication between the respective devices and components.

In the foregoing description, numerous specific details, examples, andscenarios are set forth in order to provide a more thoroughunderstanding of the present principles. It will be appreciated,however, that embodiments of the principles can be practiced withoutsuch specific details. Further, such examples and scenarios are providedfor illustration, and are not intended to limit the teachings in anyway. Those of ordinary skill in the art, with the included descriptions,should be able to implement appropriate functionality without undueexperimentation.

References in the specification to “an embodiment,” etc., indicate thatthe embodiment described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Such phrases are notnecessarily referring to the same embodiment. Further, when a particularfeature, structure, or characteristic is described in connection with anembodiment, it is believed to be within the knowledge of one skilled inthe art to effect such feature, structure, or characteristic inconnection with other embodiments whether or not explicitly indicated.

Embodiments in accordance with the teachings can be implemented inhardware, firmware, software, or any combination thereof. Embodimentsmay also be implemented as instructions stored using one or moremachine-readable media, which may be read and executed by one or moreprocessors. A machine-readable medium may include any mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device or a “‘virtual machine” running on one or morecomputing devices). For example, a machine-readable medium may includeany suitable form of volatile or non-volatile memory.

Modules, data structures, blocks, and the like are referred to as suchfor case of discussion, and are not intended to imply that any specificimplementation details are required. For example, any of the describedmodules and/or data structures may be combined or divided intosub-modules, sub-processes or other units of computer code or data asmay be required by a particular design or implementation of the contentdetection system 110. Further, references herein to rules or templatesare not meant to imply any specific implementation details. That is, thecontent detection system 110 can store rules, templates, etc. in anysuitable machine-readable format.

In the drawings, specific arrangements or orderings of schematicelements may be shown for ease of description. However, the specificordering or arrangement of such elements is not meant to imply that aparticular order or sequence of processing, or separation of processes,is required in all embodiments. In general, schematic elements used torepresent instruction blocks or modules may be implemented using anysuitable form of machine-readable instruction, and each such instructionmay be implemented using any suitable programming language, library,application-programming interface (API), and/or other softwaredevelopment tools or frameworks. Similarly, schematic elements used torepresent data or information may be implemented using any suitableelectronic arrangement or data structure. Further, some connections,relationships or associations between elements may be simplified or notshown in the drawings so as not to obscure the teachings herein.

While the foregoing is directed to embodiments in accordance with thepresent principles, other and further embodiments in accordance with theprinciples described herein may be devised without departing from thebasic scope thereof, and the scope thereof is determined by the claimsthat follow.

The invention claimed is:
 1. An automated content detection system forsemantic retrieval of multimodal content, the content detection systemcomprising one or more non-transitory machine accessible storage mediaand comprising instructions executable by one or more processors tocause a computing system to: build a semantic space by embeddingword-based document descriptions of a plurality of documents into amulti-dimensional space using a semantic embedding technique; detect aplurality of features in the multimodal content by applying featuredetection algorithms to the multimodal content; determine respectiveword-based concept descriptions for concepts identified in themultimodal content using the detected features; embed the respectiveword-based concept descriptions into the semantic space; and in responseto a content detection action, (i) embed words representative of thecontent detection action into the semantic space, (ii) automaticallydetermine, without the use of training examples, concepts in thesemantic space relevant to the content detection action based on theembedded words, and (iii) identify portions of the multimodal contentresponsive to the content detection action based on the concepts in thesemantic space determined to be relevant to the content detectionaction.
 2. The automated content detection system of claim 1, whereinthe semantic embedding technique comprises distributional semanticembedding and the content detection action comprises at least one of aconcept query or an event query.
 3. The automated content detectionsystem of claim 1, wherein the plurality of documents comprise aplurality of text-based documents comprising at least one of online newssources, online dictionaries or online encyclopedic material.
 4. Theautomated content detection system of claim 1, wherein the word-baseddocument descriptions of the plurality of documents describe concepts inthe plurality of documents.
 5. The automated content detection system ofclaim 1, wherein the respective word-based concept descriptions areembedded into the semantic space based on a location of the word-baseddocument descriptions of the plurality of documents in the semanticspace.
 6. The automated content detection system of claim 1, wherein thewords representative of the content detection action are embedded intothe semantic space based on a location of the word-based documentdescriptions of the plurality of documents in the semantic space andcorresponding word vectors for the words representative of the contentdetection action.
 7. The automated content detection system of claim 1,wherein concepts in the semantic space relevant to the content detectionaction are determined using a relative location of the word-basedconcept descriptions and the words representative of the contentdetection action in the semantic space and a similarity function.
 8. Theautomated content detection system of claim 1 wherein the computingsystem is further configured to: automatically deliver the identifiedportions of the multimodal content to a user for display.
 9. A computingdevice for zero-shot content detection of multimodal content,comprising: a processor; and a memory coupled to the processor, thememory having stored therein a multimedia content understanding module,a natural language generator module and a semantic space generatormodule comprising instructions executable by the processor to configurethe computing device to: build a semantic space by embedding word-baseddocument descriptions of a plurality of documents into amulti-dimensional space using a semantic embedding technique; detect aplurality of features in the multimodal content by applying featuredetection algorithms to the multimodal content; determine respectiveword-based concept descriptions for concepts identified in themultimodal content using the detected features; embed the respectiveword-based concept descriptions into the semantic space; and in responseto a content detection action, (i) embed words representative of thecontent detection action into the semantic space, (ii) automaticallydetermine concepts in the semantic space relevant to the contentdetection action based on the embedded words, and (iii) identifyportions of the multimodal content responsive to the content detectionaction based on the concepts in the semantic space determined to berelevant to the content detection action, without the use of trainingexamples.
 10. The computing device of claim 9, further comprising a userinterface device to enter a content detection action and a searchrequest module to enable the content detection action.
 11. The computingdevice of claim 9, further comprising at least one of an opticalcharacter recognition system to generate text transcripts of text in themultimodal content and an automatic speech recognition system togenerate text transcripts of speech in the multimodal content.
 12. Thecomputing device of claim 9, wherein the computing device is configuredto: deliver the identified portions of the multimodal content to a userfor display.
 13. A method for zero-shot content detection, comprising:building a semantic space by embedding word-based document descriptionsof a plurality of documents into a multi-dimensional space using asemantic embedding technique; detecting a plurality of features in themultimodal content by applying feature detection algorithms to themultimodal content; determining respective word-based conceptdescriptions for concepts identified in the multimodal content using thedetected features; embedding the respective word-based conceptdescriptions into the semantic space; and in response to a contentdetection action, (i) embedding words representative of the contentdetection action into the semantic space, (ii) automatically determiningconcepts in the semantic space relevant to the content detection actionbased on the embedded words, and (iii) identifying portions of themultimodal content responsive to the content detection action based onthe concepts in the semantic space determined to be relevant to thecontent detection action, without the use of training examples.
 14. Themethod of claim 13, wherein the concepts identified in the multimodalcontent include at least one of audio, video and text.
 15. The method ofclaim 13, further comprising: delivering the identified portions of themultimedia content to a user for display.